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This study examines key theoretical principles of learning-oriented evaluation for HRD programs. 
Practical guidelines for the implementation of learning-oriented evaluation are also developed based on 
the related literature. This study suggests that evaluation should be a learning process in which 
information and knowledge are socially constructed and used for individual, group, and organizational 
learning. 
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There has been growing attention on evaluation of Human Resource Development (HRD) programs in 
organizations. As the importance of optimizing human potential in the competitive business environment has been 
increasing, organization leaders need to learn more about how HRD programs operate and can be improved. 
Accordingly, HRD practitioners have paid more attention to evaluation of HRD programs and its impacts on the 
organization. On the other hand, HRD scholars have grown skeptical about the effectiveness of Kirkpatrick 
(1998)’s evaluation approach that has dominated training evaluation over the last forty years. This has resulted in an 
increasing number of studies that aim to enhance HRD evaluation theory and practice and search for an alternative 
evaluation approach to HRD programs. 

Despite this growing attention, evaluation of HRD programs in organizations, especially in the private sector, is 
still problematic. Eirst, evaluation has not been widely implemented (Swanson & Holton, 2001). Russ-fit and 
Preskill (2001) explain ten reasons why evaluation is ignored in organizations. Among the reasons, the top reason is 
organization members’ misunderstanding of evaluation purpose and role. The second reason is organization 
members’ fear of the impact of evaluation findings. In general, evaluation is negatively perceived by organization 
members. This makes evaluation difficult to implement. 

Second, evaluation approaches to HRD programs are very limited. Most of them are based on or are variations 
of Kirkpatrick (1998)’s four level evaluation approach. In spite of its strengths, such as simplicity, flexibility, and 
popularity, Kirkpatrick’ s approach has not provided a comprehensive information and understanding of how HRD 
programs work and influence organizational members. As Holton (1996) indicates, Kirkpatrick’s approach is not 
grounded in a philosophy or theory of evaluation. In addition, the majority of evaluations based on his four-level 
approach have been conducted only at level one (reaction) or level two (learning) (Dixon, 1987) 

Third, there is a significant lack of evaluation use. Preskill and Caracelli (1997) conduct a survey asking 
American Evaluation Association (AEA) members about various conceptions of evaluation use. Although their 
study does not target evaluation of HRD programs, its result shows that nonuse and misuse of evaluation are critical 
issues. In fact, many HRD scholars and practitioners have focused only on instrumental use of evaluation results 
and the evaluation results are not actually used (Russ-Eft, Atwood, & Egherman, 2002). Some researchers 
emphasize use of evaluation to justify the bottom line impact of HRD programs (Philips, 1997; Swanson, 1998). 
Yet, there is lack of research and discussion on conceptual use of evaluation and the link between evaluation and 
organizational learning. 

1 believe that these problems are driven from critical misconceptions about evaluation of HRD programs, such 
as evaluation is bad and unnecessary, evaluation means a smile sheet, and evaluation use is not important. The 
future of HRD evaluation will be shaped depending on the extent to which we can overcome the problems. The 
success of HRD evaluation relies on whether we can change these misunderstandings. In this regard, academic 
efforts for developing new evaluation approaches to HRD programs are asked in order to improve the existing HRD 
evaluation theory as well practice. New approaches should be more responsive to organization members and their 
needs and be more systemic and comprehensive in investigating the quality of HRD programs. In addition, they 
ought to contribute to increase the usability of evaluation in the private sector. 

The purpose of this paper is to investigate the development of learning-oriented evaluation for HRD programs. 
In order to achieve this, 1 first define the notion of evaluation as learning process. Key theoretical principles 
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Underlying learning-oriented evaluation are also examined based on the related literature. Further, 1 develop 
practical guidelines for successful implementation of learning-oriented evaluation. Learning-oriented evaluation, 
which aims at fostering different types and levels of learning through evaluation, can change negative conceptions of 
evaluation to a learning opportunity. Evaluators as well as stakeholders can have opportunities to broaden and 
deepen their knowledge and understanding of HRD programs from this kind of evaluation approach. Learning- 
oriented evaluation also can help organizational members consider evaluation as ongoing learning experiences rather 
than as a one-time event. Moreover, it can produce useful and comprehensive information of the program being 
evaluated. So, evaluators and stakeholders can develop diverse ways to enhance evaluation use and to improve 
HRD programs. Evaluation should be an opportunity and catalyst to learning in organizations. 

Evaluation as Learning Process 

Several evaluation researchers, such as Lee Cronbach, Gretchen Rossi, John Owen, Hallie Preskill, and Rosalie 
Torres, have emphasized evaluation as a process of fostering learning. Although there are variations in explaining 
the learning dimension of evaluation among them, their major focus is on helping evaluators as well as stakeholders 
learn from evaluation, so they can deepen their understanding, obtain new insights, take visible actions, and improve 
quality of an evaluand - product, program or process being evaluated. 

Cronbach and his colleagues contend that “evaluation is a process by which society learns about itself’ 
(Cronbach et ah, 1981, p.2). According to them, evaluation should be a learning opportunity that enables 
stakeholders to know better about the program, understand multiple perspectives, and thus make wise choices for the 
program improvement. In doing this, an evaluator need to play a role as an educator that facilitates sharing as well 
as providing information. Although their main concern is to help members of the “policy-shaping community”, 
their contention provides at least two important points of view for evaluation of HRD programs. Eirst, they suggest 
a different view on the role of an evaluator from the traditional one. According to them, it is not an evaluator’s task 
to decide whether the program is good enough. Instead, the evaluator provides information which stakeholders may 
want to take into account in forming their judgments. Second, Cronbach et al. (1981) believe that things can be 
changed because of stakeholders’ learning through evaluation. Evaluation can directly or indirectly change 
stakeholders’ understanding and behavior, whether it is intended or unintended, throughout evaluation. 
Accumulations of these changes can affect procedure and culture of the program and other parts of organization 
processes, so in the long run, the organization can be changed. 

The concept of evaluation as learning process is further explained by Preskill and Torres (1999), who suggest 
the notion of evaluative inquiry as a new form of evaluation for learning. Since organizations have experiencing 
continuous and dramatic changes due to advanced technology, workforce diversity, and globalization, continuous 
learning and development becomes the key for organizations to survive (Dixon, 1992; Ruona, Lynham, & 
Chermack, 2003). Under consideration of this feature. Preskill and Torres (1999) emphasize that evaluation should 
be integrated into daily work practice in organizations and thus it ought to be an ongoing inquiry for learning. In 
this sense, they introduce evaluative inquiry, which encourages organization members to have chances of exploring, 
reflecting, questioning and discussing critical organization issues based on the evaluation logic and process. They 
believe that this will bring out enhanced organization capacity for continuous learning as well as personal and 
professional development of organization members. 

Learning through evaluation can occur at individual, group, and organizational level. Individual learning can 
occur when individuals are able to alter their perceptions and knowledge of the program from the evaluation process 
and findings, achieve new insights, and take actions based on the insights (Rossman & Rallis, 2000). Learning from 
evaluation at the group level can take place when individuals share their experiences, knowledge, and insights 
through dialogue and participate in collaborative learning situations. Since evaluation involves, in most cases, 
different groups of stakeholders, it is important to provide chances of communicating across the groups and 
understanding diverse perspectives of the program that will increase the potential of stakeholders for learning. 
Organizational learning from evaluation can occur when individuals and groups disseminate their learning and 
knowledge obtained from evaluation throughout the organization. This will lead to organizational change and 
development. 

Key Theoretical Bases of Learning-Oriented Evaluation 

Learning-oriented evaluation is concerned with developing the potential of stakeholders as well as evaluators by 
providing opportunities to learn and stimulating social interactions among stakeholders and evaluators in a given 
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context. In this section, I examine theoretical assumptions of learning-oriented evaluation in terms of its claims on 
knowledge construction, evaluation use, and organizational influence. 

Knowledge Construction 

Most researchers who advocate learning-oriented evaluation emphasize participation and ownership of 
stakeholders in creating information and knowledge useful. This assertion is based on the belief that knowledge and 
values are socially constructed and transited rather than exist independently from the social world waiting to be 
revealed by technically expert evaluators. Therefore, stakeholders should be active participants in construction of 
knowledge through evaluation and use of the knowledge in their work. In fact, HRD programs involve adult 
learners in the workplace. Compared to children, adults are more self-directed and learn from their experiences. 
They tend to have their own willingness, motivation, intention, and styles for learning. These characteristics of adult 
learners enable learning-oriented evaluation of HRD programs to be a platform for stakeholders to construct and use 
their own knowledge. 

From the constructivist view, learning is also a process of meaning making (Preskill & Torres, 2000; Rossman 
& Rallis, 2000). Individuals and groups learn by understanding and making meanings of their experiences. In this 
sense, learning from evaluation is socially situated and is mediated through stakeholders’ previous knowledge and 
experiences. Constructions of meanings are influenced by specific historical, political, geographical, and cultural 
practice and discourses in a given context (Greene, 2000). These constructions are multiple, contingent, and 
contextual. It is, therefore, important through learning-oriented evaluation to involve and discover multiple 
constituencies each representing different perspectives. 

Evaluation Use 

Although the concept of evaluation use is still diverging and expanding (Shulha & Cousins, 1997), there are 
mainly three different kinds of evaluation use based on the purposes they serve. They are instrumental, conceptual, 
and symbolic use (Weiss, 1998). Learning-oriented evaluation can be employed for instrumental, conceptual, and/or 
symbolic purpose. 

Instrumental use of evaluation aims at providing information for decision makers. Evaluation findings and 
recommendations are used to give feedback into decision making processes for the program. According to Weiss 
(1998), this kind of use is fairly common under three conditions. First, the application of evaluation findings is 
relatively non-controversial. Second, the change that occurs as a result of evaluation is relatively small-scale. 
Lastly, internal and external environments of the program are relatively stable without dramatic change in 
leadership, fund, and stakeholders. If evaluators and stakeholders learn from findings of learning-oriented 
evaluation and make informed decisions regarding operation, continuation, or improvement of the program, this 
evaluation is instrumentally used to produce information for decision-making. 

Conceptual use of evaluation is concerned with changing and developing, through evaluation, stakeholders’ 
understanding of what the program is and does. Many aspects of learning-oriented evaluation are connected to this 
kind of use, because learning-oriented evaluation focuses primarily on the evaluation process and conceptual change 
(learning) of the people who are involved in evaluation. In fact, stakeholders can obtain new ideas and insights 
about the program from evaluation findings. However, they can learn even more about merits and limitations of the 
program and possible directions for action by engaging in the evaluation process. The notion of Patton (1997)’s 
process use is similar to this kind of use. Patton (1997) highlights evaluators as well as stakeholders’ changes as 
participating in the evaluation process. 

Process use refers to and is indicated by individual changes in thinking and behavior, and program 
or organizational changes in procedures and culture, that occur among those involved in 
evaluation as a result of the learning that occurs during the evaluation process, (p. 90) 

Preskill and Torres (2000) describe this kind of use as a form of individual, team, and organizational learning. They 
suggest that when individuals participate in the evaluation process that is collaborative and guided by dialogue, 
reflection, and critical inquiry, learning occurs not only at the individual level but also at the group and 
organizational level. 

Symbolic or political use of evaluation is intended to strengthen one’s positive or negative opinions about the 
program (Marra, 2003). For example, program managers know what is not working and how problems can be 
solved. Thus, they use evaluation to legitimize their position and gain supports from other members. Moreover, the 
very existence of evaluation can influence organization members. For example, when learning-oriented evaluation 
is carried out in the organization that never or little has employed this type of evaluation approach, the presence of 
learning-oriented evaluation creates a level of interests among organization members and makes them be aware of 
needs for learning and change. 
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Organizational Influence 

Evaluation is performed within an organizational context. Learning through evaluation is critically influenced 
by the organization’s goals, systems, structures, and culture. Rogers and Hough (1995) claim that focus, methods, 
and management of evaluation should reflect realistic assumptions about how the organization works. Without 
understanding of how individual, group, and organizational learning occurs in the organization, learning-oriented 
evaluation cannot be successfully implemented and its benefits will be significantly reduced. 

In addition, several researchers stress that evaluation capacity building is critical in order to “continuously 
create and sustain overall organizational processes that make quality evaluation and its uses routine” (Compton, 
Baizerman, & Stockdil, p.l). The success of learning-oriented evaluation influences and is influenced by the 
organization’s infrastructure that supports continuous learning. Therefore, structures, policies, procedures of the 
organization need to incorporate and maintain learning through evaluation. Desirable organization systems, 
structures, and culture make it easy to share evaluation information and make it possible to work collaboratively 
among different stakeholder groups. These organization features also allow organization members to try new things 
and to consider mistakes as opportunities for learning. It is obvious that the organization including such desirable 
systems, structures, and culture is motivated to provide organizational members financial, technological, and human 
resources for learning. 

Moreover, organization members should understand the importance and potential of learning from evaluation. 
Especially, leaderships at all levels of the organization ought to support and consider evaluation as a means for 
individual, group, and organizational learning. Organization leaders at top positions need to provide a vision for 
learning through evaluation and encourage organization members to value and prioritize evaluation. Leaders at the 
program levels need to realize their responsibility for continuous program improvement and appreciate partnership 
with an evaluator in creating evaluation information. 

Because effective use of learning-oriented evaluation sometimes requires organizational changes, there is an 
issue of how to deal with organization members’ resistance to changes. Organization members tend to defend the 
status quo and have negative expectations of a new evaluation approach. According to Taut and Branus (2003), 
organization members have fears that they can be criticized or lose their jobs as a result of evaluation. These fears 
are often rooted in lack of trust among organization members and misunderstanding of what learning-oriented 
evaluation can be and do for them. Therefore, it is essential that the development of learning-oriented evaluation 
should be based on shared trust, value, and belief that evaluation can be a catalyst for individual, group, and 
organizational learning. 

Key Practical Guidelines for Learning-Centered Evaluation 

Learning-oriented evaluation is somewhat similar to other evaluation approaches that also emphasize learning as an 
outcome of evaluation, such as participatory, stakeholder-focused, empowerment, or utilization-oriented approach. 
To varying degrees, these approaches position an evaluator as a facilitator of learning from evaluation in which 
stakeholders learn about themselves as well as the program. However, learning-oriented evaluation for HRD 
programs is distinguishable from other approaches in that it focuses on increasing the potential of stakeholders and 
contributing to achieve organizational learning. In this section, 1 explain crucial strategies for the practice of 
learning-oriented evaluation. 1 develop one strategy for each evaluation stage that should be a focus or at least be 
gained attention. 

Purpose and Audience 

Identify multiple stakeholders and their information (learning) needs. The main concern of learning-oriented 
evaluation is to enlighten people involved. Therefore, it is important to include diverse audiences and understand 
what they want to know and learn through evaluation. Different purposes can be developed depending on the 
context- the program or intervention being evaluated, problems identified, stakeholders engaged, a type of the 
organization, and so on. Lor instance, one purpose of learning-oriented evaluation may be to provide information 
for decisions needed to be made regarding the program’s improvement and expansion. Anther purpose may include 
gaining a better understanding of the program’s effects on different groups. Whatever the purpose of the evaluation 
may be, an evaluator needs to ensure that the purpose(s) of the evaluation is formed based on diverse stakeholders’ 
needs for learning and action. Diverse audiences, such as organization leaders, program directors, staff, trainers, 
trainees, and community members can be identified. Lrom the beginning of the evaluation, the evaluator needs to 
facilitate the stakeholders to be informed of different issues and to priority them. Thus, they can decide what they 
need to know and what purpose is appropriate and feasible in a given situation. The purpose(s) of the evaluation 
chosen should be clearly stated and be effectively shared with the stakeholders. 
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Questions 

Develop a set of questions in negotiations with stakeholders. Evaluation questions are very important, since 
these determine the boundary as well as the focus of the evaluation. Evaluation questions also help to further define 
the purpose of the evaluation and can influence decisions about evaluation design and methods for data collection. 
The evaluator and stakeholders may start from overarching and broad questions. Eor instance, what do or don’t we 
know about the program?. What do we need to know?. How does the program work?. How does the program 
achieve its goals?, and so on. It may be better to develop open-ended questions, because open-ended questions 
likely are more responsive to change of stakeholders’ needs as well as evaluation environments and produce rich 
information. 

The process of developing evaluation questions calls for intense negotiations with stakeholders. Learning- 
oriented evaluation seeks to involve diverse stakeholders. So, it is not easy to come to agreement on a set of 
questions across different groups. While certain things are crucial to some stakeholders, others may not agree with 
that. Therefore, the evaluator needs to encourage stakeholders to explore their assumptions and values underpinning 
their support of certain questions. In this case, stakeholders can articulate why particular questions are more 
important and relevant to the purpose of the evaluation. Eurthermore, they can understand what others think and 
expect from the evaluation. Through these processes, the evaluator and stakeholders can develop deeper 
understanding and new insights about the program (conceptual use). This also increases the likelihood that 
meaningful and useful data will be obtained as carrying out the evaluation (Cousins, Donohue, & Bloom, 1996). In 
addition, stakeholders who attained deeper knowledge and understanding about the program can influence other 
organizational members’ understanding of the program and the people involved. Both the evaluator and stakeholders 
need to make key inputs in the negotiations for developing evaluation questions. Eor example, stakeholders can 
bring knowledge of the program and the organization, while the evaluator is able to indicate limits to the evaluation 
given resources available. 

Design and Methods 

Select the most relevant, effective, and appropriate design and methods for data collection. The main focus in 
selecting proper evaluation design and methods is on understanding which design and methods will produce the best 
possible data to adequately answer evaluation questions. According to Preskill and Torres (1999), the evaluator and 
stakeholders can consider at least four factors in making decisions regarding evaluation design and methods. 

• Preferred methods of those experienced in data collection and analysis activities. 

• Prior experiences of team members with data collection and analysis activities. 

• Unexamined notions, perceptions, or prior knowledge about the organization’s culture 

• Perceived methodological, time, or other logistical constraints (p.lOl). 

The evaluator should not employ a particular design or methods without having the expertise to use them. Also, 
resources and time available for data collection need to be checked before determining a design and methods. In 
consultation with stakeholders, the evaluator can make decisions about an overall evaluation design - experiment, 
quasi-experiment, systems analysis, case study, or participatory process, so on - and specific data collection and 
analysis methods - questionnaires, standardized measures, interviews, and observations, so on. Many researchers 
recommend that use of multiple information sources and methods is more likely to generate evaluation data that 
represent accurate and true indications of the program (Patton, 1997; Russ-Eft & Preskill, 2000). 

Criteria for Judging Program Quality 

Clarify stakeholders’ assumptions of the program quality. There is often confusion or difference among diverse 
stakeholders about evaluation criteria of the program should be. Even though they agree on a certain constitution of 
the program quality, for example effectiveness, or productivity, they may have different assumptions about what 
effectiveness or productivity means. Therefore, Evaluation criteria need to be defined based on shared opinions and 
similar assumptions of the program quality across different groups of stakeholders. 

Developing evaluation criteria is a contextually sensitive matter, so that there is no single and objective standard 
out there to be used for HRD programs. Evaluation criteria should be mutually defined among the evaluator and 
stakeholders in a given context. One of the problems that HRD evaluation has faced is that “satisfaction” of 
customers or trainees has been overused to determine merit and worth of HRD programs. Due to loosely employing 
Kirkpatrick’s evaluation approach, satisfaction or reaction has become the key criteria to judge HRD programs even 
when satisfaction is not actually relevant to the quality of the program being evaluated. 
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Reporting 

Share final findings with all stakeholders. Some researchers emphasize on reporting to a certain group of 
stakeholders, for example intended users (Patton, 1997), influential program staff (Owen, Lamber, & Stringet, 1994; 
Rossman & Rallis, 2000), premising that use of evaluation findings can be ensured by them. However, when 
considering the importance of information as a major source for learning, evaluation information should be reported 
to and shared with all of the stakeholders. This does not mean that every stakeholder needs to have a same type of 
report. Various versions of reports — short or long, electric or printed — can be used. Even though a thick and 
comprehensive report can describe findings, recommendations, and implications in details, it is not always necessary 
for all stakeholders to receive such report, and it takes high cost to produce it. Mattson (2003) studies managers’ 
perceptions of different types of evaluation reports in terms of their usefulness for decision making. The study 
shows that managers tend to use evaluation information more when it is reported in a way that is shorter in length 
and higher in comprehensibility. In addition to written evaluation reports, verbal presentations can be used for 
reporting evaluation findings. Sometimes, presentations can be more effective in informing and stimulating 
stakeholders for learning and action, because the evaluator and stakeholders can have chances to directly discuss 
evaluation information, ask questions, and voice reactions or concerns. 

Participation 

Create genuine forms of stakeholder participation. Learning-oriented evaluation recognizes the importance of 
including multiple stakeholders’ views. In this sense, the evaluator needs to provide opportunities over time for 
diverse stakeholders to better understand benefits and limitations of the program by engaging in the evaluation 
process. Stakeholders are not only relevant and appropriate sources of evaluation information, but they also play a 
role in creating evaluation practice as well (Cousins, Donohue, & Bloom, 1996; Greene, 1988). Therefore, the 
evaluator should work collaboratively with stakeholders in defining and developing many aspects of the evaluation. 
Diverse stakeholders’ participation increases the likelihood of their commitment to evaluation and use of evaluation 
findings. Stakeholders’ participation also increases the likelihood that individual, group, and organizational learning 
take place through collaborative group efforts. Even though it is necessary that multiple stakeholder groups 
participate in the evaluation process, it is often difficult to achieve this element within realities of the program 
operations. 

Use 

Provide ongoing support for stakeholders to integrate learning into their daily work. If learning-oriented 
evaluation is successfully implemented as intended, use should not be a problem since use is integral to learning 
through evaluation. In other words, if individual, group, and organization learning occur during the evaluation 
process, this means that intended use of evaluation is achieved. As already mentioned, in addition to use of 
evaluation results, there is conceptual or process use that is important and needs to obtain more attention from HRD 
professionals. Again, stakeholders’ participation in the evaluation process enhances learning and thus increases 
evaluation use. Moreover, use of evaluation is not the solo responsibility of the evaluator, because the evaluator and 
stakeholders are partners and co-producers in generating evaluation information as well as forming evaluation 
practice. Therefore, a collective commitment is required to apply learning into the program operations and 
outcomes. 

Rather than a one time event, evaluation needs to be a force for continuing efforts for learning and action. As a 
means of integrating learning from evaluation into daily work. Preskill and Torres (1999) propose that the evaluator 
and stakeholders collaboratively work to develop an action plan for implementation of recommendations based on 
evaluation findings. According to Preskill and Torres (1999), after sharing evaluation information with 
stakeholders, the evaluator should encourage stakeholders to consider various action alternatives, choose among 
them, and develop an action plan to implement the alternative(s) selected. To maximize use of evaluation findings, 
the evaluator can work with stakeholders to implement program or organizational changes recommended by the 
evaluation. However, these collaborative endeavors need to be based on other organizational members’ trust and 
support. 

Contribution to HRD Research and Practice 



As the importance of evaluation in organizations has increased, there have been academic endeavors that suggest 
alternative approaches to evaluation of HRD programs. The notion of evaluation for learning has influenced HRD 
scholars and practitioners to expand their perception of evaluation use and see a clearer link between evaluation and 
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learning in organizations. However, there are still increasing needs for developing and articulating learning-oriented 
evaluation in which information and knowledge are socially constructed, and learning is facilitated by people and 
their social interactions. There is also a little theoretical discussion and practical applications of how this kind of 
evaluation approach helps to better develop organization members and enhance the quality of HRD programs. In this 
regard, this study attempt to provide conceptual as well as practical information needed when developing an 
evaluation for learning. Future research may need to take a deeper look at theoretical background of learning- 
oriented evaluation. Also, more empirical studies should be conducted to support evaluation as a means of 
individual, group, and organizational learning. In addition, it is necessary to investigate how and when the 
organization capacity for learning influences the development of learning-oriented evaluation for HRD programs. 
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